<html>
<head><meta charset="utf-8"><title>How to debug a memory leak · general · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/index.html">general</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html">How to debug a memory leak</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="199954287"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199954287" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199954287">(Jun 06 2020 at 03:43)</a>:</h4>
<p>I have a language server implemented in Rust, and I have noticed that it consumes far more memory than it is expected to, especially over time and use. I don't know if the problem is a memory leak in the strict sense of leaked allocated memory (which is possible as the program contains a lisp interpreter using Arc's behind the scene and perhaps Arc cycles are getting away) or whether one of my data structures is simply holding on to references it doesn't need to. I have no idea where in the program the issue is, I have only a 1000-km view of the situation. What techniques can I use to narrow down the problem to something more actionable?</p>
<p>I have tried using <a href="https://github.com/japaric/rust-san">rust-san</a> but because it is a language server the provided interface (using environment variables to start it and communicating on standard out) conflicts with testing in situ (where it is run by the editor using a fixed command string and IO is used for communication with the editor).</p>



<a name="199954390"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199954390" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199954390">(Jun 06 2020 at 03:47)</a>:</h4>
<p>One thing which would be great is a way to take a "census" of the owned allocated memory, similar to the view a garbage collector would have, perhaps using some macros? Is there a crate out there for this kind of thing?</p>



<a name="199957424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199957424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199957424">(Jun 06 2020 at 05:34)</a>:</h4>
<p>There's <a href="https://valgrind.org/docs/manual/dh-manual.html">DHAT</a>, but it might be a bit low-level for you. You don't get a report aggregated by type like in a managed language, but by allocation stack trace.</p>



<a name="199957469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199957469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199957469">(Jun 06 2020 at 05:35)</a>:</h4>
<blockquote>
<p>because it is a language server the provided interface (using environment variables to start it and communicating on standard out) conflicts with testing in situ (where it is run by the editor using a fixed command string and IO is used for communication with the editor)</p>
</blockquote>
<p>Actually, being a language server might be a great thing. You could record the LSP trace (use it for a couple of hours) and play it back to a fresh instance of the server in a matter of seconds. You could even add it to CI to make sure you don't regress performance or memory usage.</p>



<a name="199958094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958094">(Jun 06 2020 at 05:55)</a>:</h4>
<p>Is there a way to run valgrind/DHAT/rust-san in a way so that it outputs to a file that I specify in the server code itself (rather than using command line inputs)?</p>



<a name="199958140"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958140" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958140">(Jun 06 2020 at 05:56)</a>:</h4>
<p>These set ups all seem to assume that stdout is free for printing tracing info but this immediately crashes the editor connection because it's expecting JSON</p>



<a name="199958151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958151">(Jun 06 2020 at 05:56)</a>:</h4>
<p>I don't think so. You could also write a wrapper script that launches <code>valgrind</code> with the parameters you expect and use it as the LSP server</p>



<a name="199958156"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958156" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958156">(Jun 06 2020 at 05:56)</a>:</h4>
<p>I think they log to <code>stderr</code>, not <code>stdout</code>?</p>



<a name="199958202"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958202" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958202">(Jun 06 2020 at 05:57)</a>:</h4>
<p>that's a good point, I <em>think</em> that vscode should forward stderr output to its own output console rather than crash and burn</p>



<a name="199958246"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958246" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958246">(Jun 06 2020 at 05:58)</a>:</h4>
<p>It does, see <a href="https://github.com/rust-analyzer/rust-analyzer/tree/master/docs/dev#logging">https://github.com/rust-analyzer/rust-analyzer/tree/master/docs/dev#logging</a></p>



<a name="199958247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958247">(Jun 06 2020 at 05:58)</a>:</h4>
<p>and with your unit test idea it doesn't matter since the simulated client isn't paying attention to what the server says</p>



<a name="199958264"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958264" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958264">(Jun 06 2020 at 05:59)</a>:</h4>
<p>Yeah. You could record and compare it to a known good output, but..</p>



<a name="199958268"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958268" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958268">(Jun 06 2020 at 05:59)</a>:</h4>
<p>but I tried enabling leak check and noticed no difference, no output or anything and I haven't yet figured out whether I didn't turn it on right or whether I actually have no leaks</p>



<a name="199958313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958313">(Jun 06 2020 at 06:00)</a>:</h4>
<p>If you don't call <code>mem::forget</code> or have leaks in some native libraries, you probably don't have leaks <em>per se</em> (though you can still e.g. make reference cycles).</p>



<a name="199958335"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958335" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958335">(Jun 06 2020 at 06:00)</a>:</h4>
<p>well like I said, arc cycles are a possibility, although it should only matter if the language source I'm reading does some funny business</p>



<a name="199958355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958355">(Jun 06 2020 at 06:01)</a>:</h4>
<p>originally I estimated that this is not a major concern and I have not yet had the facility to test that estimation</p>



<a name="199958401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958401">(Jun 06 2020 at 06:02)</a>:</h4>
<p>Do you support those reader macros things -- how are they called? The <code>#1=(1 2 . #1#)</code> syntax.</p>



<a name="199958413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958413">(Jun 06 2020 at 06:03)</a>:</h4>
<p>It has mutable cells and function closures, so you can tie up loops if you want to but it would be weird code</p>



<a name="199958424"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958424" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958424">(Jun 06 2020 at 06:03)</a>:</h4>
<p>not directly but you can simulate them with refs</p>



<a name="199958475"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958475" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958475">(Jun 06 2020 at 06:04)</a>:</h4>
<p>But only at run-time, right? And so your LSP server won't evaluate them?</p>



<a name="199958482"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958482" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958482">(Jun 06 2020 at 06:05)</a>:</h4>
<p>The code is evaluated on the spot</p>



<a name="199958485"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958485" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958485">(Jun 06 2020 at 06:05)</a>:</h4>
<p>The language isn't actually lisp, lisp is the macro language for my language</p>



<a name="199958493"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958493" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958493">(Jun 06 2020 at 06:05)</a>:</h4>
<p>Ah</p>



<a name="199958550"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958550" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958550">(Jun 06 2020 at 06:06)</a>:</h4>
<p>I suspect the problem is that the server is holding on to environment data for all old versions of a file, but it seems like most attempts to profile it would require large code changes</p>



<a name="199958603"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958603" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958603">(Jun 06 2020 at 06:08)</a>:</h4>
<p>Or small changes repeated a lot of times. Like removing and adding back code or something like that. You could probably edit the LSP trace to get e.g. a million of those.</p>



<a name="199958611"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958611" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958611">(Jun 06 2020 at 06:09)</a>:</h4>
<p>in theory it should maintain constant memory over a workload like that</p>



<a name="199958658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958658">(Jun 06 2020 at 06:10)</a>:</h4>
<p>Is there an easy way to obtain an LSP trace? I guess the server can pass what it sees to a file</p>



<a name="199958660"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958660" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958660">(Jun 06 2020 at 06:10)</a>:</h4>
<p>VS Code can also produce a trace, there's a special <code>"extension-name.trace.server": "verbose"</code> setting, but it adds some timestamps or such to every request. I guess logging from the server would be the easiest? Another trick I used once was to write a small program that spawns the server and logs its stdin/stdout to files.</p>



<a name="199958725"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958725" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Mario Carneiro <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958725">(Jun 06 2020 at 06:13)</a>:</h4>
<p>ah, something like <code>server.sh</code> containing <code>tee lsp.log | server</code></p>



<a name="199958772"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/122651-general/topic/How%20to%20debug%20a%20memory%20leak/near/199958772" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/122651-general/topic/How.20to.20debug.20a.20memory.20leak.html#199958772">(Jun 06 2020 at 06:14)</a>:</h4>
<p>Sigh. I wrote some Rust code for it. <code>tee stdin.log | server | tee stdout.log</code> might have done the trick.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>